Hands-on laboratory and demonstration equipment with a hybrid virtual/augmented environment, along with their methods of use

ABSTRACT

Systems and methods for utilizing laboratory equipment in a hybrid reality environment or augmented virtual reality environment, are contemplated herein and include: at least one piece of laboratory equipment having at least one feature, wherein the at least one piece of laboratory equipment is tracked; a tracking module for tracking a position and an orientation of the at least one piece of tracked laboratory equipment; a virtual model, stored in a memory, comprising at least one 3-D virtual representation of the at least one feature of the at least one piece of laboratory equipment; and an experimentation module. A piece of laboratory equipment for use in a hybrid reality environment or augmented virtual reality environment is also included that comprises at least one piece of laboratory equipment having at least one feature, wherein the at least one piece of laboratory equipment is tracked; at least one marker that is coupled with the at least one piece of laboratory equipment; and a tracking module for tracking a position and an orientation of the at least one piece of tracked laboratory equipment, wherein the tracking module accesses the at least one marker.

This United States Utility Patent Application claims priority to U.S.Provisional Patent Application Ser. No. 62/657,771, which was filed onApr. 14, 2018 and is entitled “Hands On Laboratory and DemonstrationEquipment with a Hybrid Virtual Environment, Along With Their Methods ofUse”, which is also commonly-owned and incorporated in its entirety byreference.

FIELD OF THE SUBJECT MATTER

The field of the subject matter is laboratory and demonstrationequipment that is designed to be hands on, wherein the full experienceis demonstrated and utilized in both a live action environment and ahybrid virtual or an augmented environment.

DETAILED DESCRIPTION Background

Learning is maximized when theory and practice are seamlessly integratedand when spatially and temporally coordinated visual, auditory, andtactile sensory learning experiences reinforce each other. Nowhere isthis more evident than in traditional hands-on laboratory classes,industry internships, and mentored research projects where studentsapply theoretical concepts, carry out experiments/procedures, andanalyze data in an active, guided, and often open-ended manner usingtheir eyes, ears, and hands. Despite being prized by STEM students,employers, and academic research institutions such experiences arehighly resource intensive and thus a challenge to offer in a scalablemanner. For example, dangerous, long, complicated, or expensiveexperiments are often difficult to offer.

Despite the fact that both learning in general and the scientific methodin particular involve iterative refining of our conceptual understandingof our environment, in traditional lab classes students are often notfree to make mistakes, redo experiments, or iteratively refinehypotheses to attain mastery over techniques and concepts. The longhours and lack of scheduling flexibility in traditional face-to-facelabs also reduces STEM participation by low-income and underrepresentedminority students because they often work while in school. On-demanddynamic visualization of the molecular or microscopic basis for amacroscopic laboratory observation is often absent in traditionaltextbooks, lectures, and lab activities. Finally, lab instructors,industry trainers, and research mentors have limited time with students;this prevents them from correcting the most nuanced types of mistakesthat students often make when learning new methods.

Universities and graduate schools who provide STEM training to studentsbalance several goals: first, they must provide an environment where keylab skills can be practiced and ultimately learned; second, they mustkeep overall costs as low as possible, while providing the necessaryresources, and third, they need to ensure that the students and the labenvironment are as focused and safe as possible.

Conventional lab training involves students working with actualequipment, such as beakers, pipettes, chemicals, heat sources, and othermaterials that may be unpredictable in the hands of inexperiencedscience students. Depending on the experiments, there may be safetyissues. But, at the very least, experiments may need to be run more thanonce by a student or team, supplies can be damaged or can break, and thecosts of materials can be expensive.

There are some teaching environments that are abandoning working withactual equipment and hands-on experiments and opting instead for avirtual environment or an animated environment, where the studentsessentially watch the experiments take place and may participate in thelab experience by moving their hands, eyes, voice, or head. In somevirtual systems, the students hand movement operates a robot or roboticdevice that is actually doing the work.

There is a lot of work in this field in the area of surgery simulationand training or the use of virtual/augmented reality in actualsurgeries. For example, US Patent 9251721 discloses the use of virtualreality coupled with physical models to simulate surgical procedures.The patent and its related patents and patent applications discuss thatthe physical model of a subject object needs to be penetrable through anexternal or internal surface. The patent also discloses that one of theobjects of the invention is to allow the user to “see” through the useof a virtual model parts of the body that are either internal or “blind”to the user (“internal or hidden features of the object”) through normalsurgical processes. In addition, the patent discloses a “virtualrepresentation” that is presented to the user that contains the virtualrepresentations of the internal organs of the body, the surgicalinstruments, and any additional items not visible to the user inordinary circumstances. There is still a need to utilize augmentedreality viewing/simulations in lab settings.

Nearly 95% of a biochemistry/molecular biology researcher's time at thebench is spent interfacing with test-tubes, their pipetteman, tip racks,and stocks of various solutions/reagents. If one could provide thesensory experience of handling these lab tools, while allowing them tocross the boundary between the real and virtual world of aninstructional simulation, an entirely new class of truly “hands-on”hybrid virtual labs would become accessible. Students would come closeto getting a fully hands-on scientific learning experience without anyof the traditional resource constraints that prevent universities andother schools from providing all students with all the resources theymight want for a given lab course.

For example, it would be advantageous if science lecture and lab coursescould be redesigned to use virtual lab modules effectively to eitherimprove student engagement with the course content and/or to reducebottlenecks/resource constraints that typically prevent institutionsfrom offering more class sections. However, a major issue withconventional virtual reality tools and instructional lab modules is thelack of authentic tactile sensory experiences for the user. Virtualreality (VR), augmented reality (AR), and mixed reality (MR)applications have progressed rapidly in the past 5 years. However,although there are now many games, experiences, and hardware controllersfor non-instructional applications, the use of VR, AR, and MR ineducational settings has been limited. In higher education instructionallaboratories where students must work with the actual physical tools oftheir work, this is mainly because virtual labs don't offer an authentic“hands-on” experience to these student users.

One route of providing the hands-on experience that companies aredeveloping is the use of haptic gloves associated software and similartechnology, where a student puts on a glove laden with sensors andactuators that sense the users hand position and create a “passable”virtual representation of the users hand within the virtual world.Real-time software then calculates a set of resistive forces which whenapplied onto the user's hand would authentically recreate the tactileresistance that the virtual world should induce upon the user at a givenmoment in the virtual reality experience. These forces are then passedto the actuators on the haptic glove in the real world to recreate thatresistance and tactile sensory experience for the user. When the studentmoves his or her hands with the gloves on, the student sees aninteractive experience through a virtual reality portal. While this is aterrific advancement in the broader field of augmented/virtual reality,and may provide—at the coarsest levels an somewhat authentic replicationof the tactile resistance that users might experience when performinglaboratory experiments (e.g. grasping a graduated cylinder) it doesn'tprovide students with finely-grained tactile sensory feedback from thevirtual world or the ability to really feel the tools they will beusing—how those tools are manipulated, how solutions are transferred,how test tubes are opened, culture plates are manipulated, howcomponents are mixed/added to one another, etc. The “feel” of tools in astudent's hand is a necessary component of learning how to work in a labenvironment.

Schell Games has developed a virtual lab called HoloLab Champions thatsimulates several experiments and enables the user to develop laboratoryskills and learn best practices. However, the user has to hold apaddle-like controller in his or her hands in order to interact with thevirtual world (i.e. pick up and manipulate items or use equipment).There is no authentic tactile sensation of picking up a beaker orweighing out a powder. The user does not pick up beakers or interfacewith other equipment by hand, but instead uses the controller. Prior ArtFIG. 1 shows a brief comparison of several tracking approaches which canhelp resolve this problem. In particular, tracking technologies allowthe position and state of real objects (e.g. fake guns, golf clubs,baseball bats, ping pong rackets, etc.) to be detected and passed into avirtual reality environment even as the user is holding and manipulatingthose objects in actual reality.

To date, there isn't anything of this sort adapted specifically fortraining undergraduate students in science education by passing realscientific laboratory tools (e.g. pipettemen, test tubes, graduatedcylinders, and other equipment) that are held by a user and trackedusing any of the above-listed tracking technologies into the virtualworld. Such an approach which we call “virtualization of lab tools” hasthe advantage of providing fine-grained and completely realistic tactilesensory feedback to the user directly from the real physical scientifictool they are holding, while also allowing the physical scientific toolto pass into the virtual world and properly interact with all theexclusively-virtual instruments and reagents of the virtual realityexperience (e.g. solutions to be pipetted from tube to tube, etc).

A common argument against replacing traditional science labs withvirtual labs is that virtual labs can't provide certain essentialtactile sensory feedback to students. While there are clearly somepractical laboratory skills that can only be acquired by holding,manipulating, and gaining physical familiarity with laboratory tools,there are currently few studies which clearly identify what these skillsare and how tool manipulation facilitates learning. This stems largelyfrom the challenges involved in deconvoluting the impact of tactilesensory feedback from all the other differences that exist betweenvirtual and traditional lab experiences. The methods developed here willenable controlled studies (ie. VR with hands-on lab tools vs. VR onlyw/o lab tools vs. traditional hands-on lab) which can more directlyaddress this important issue and define exactly how and when tactilesensory feedback is essential to learning in the sciences.

To address the shortcomings of conventional technology, a hybrid virtualstudent experience has been developed where positional trackingmarkers/sensors are placed on or built into the laboratory items andtools that need to be actually manipulated by the user (e.g. pipettemenand sample tube) in order to preserve the real feel of the manipulatedobject while eliminating or minimizing the costs associated withproviding authentic haptic force feedback to the user. In an idealizeduse case, labs and schools who cannot afford to replace existingnon-functional equipment can utilize tracking markers/sensors tovirtualize the non-functional lab equipment and make it functional forinstructional purposes within a virtual learning environment withouthaving to actually fix or maintain it.

SUMMARY OF THE SUBJECT MATTER

Systems and methods for utilizing laboratory equipment in a hybridreality environment, are contemplated herein and include: at least onepiece of laboratory equipment having at least one feature, wherein theat least one piece of laboratory equipment is tracked; a tracking modulefor tracking a position and an orientation of the at least one piece oftracked laboratory equipment; a virtual model, stored in a memory,comprising at least one 3-D virtual representation of the at least onefeature of the at least one piece of laboratory equipment; anexperimentation module that: accesses the virtual model stored in thememory; receives input from the tracking module regarding the positionand the orientation of the at least one piece of tracked laboratoryequipment; determines, based on the input from the tracking module, acorresponding interaction between the at least one piece of laboratoryequipment and at least one additional piece of laboratory equipment, atleast a portion or part of a user, or a combination thereof; determinesa consequence of the corresponding interaction; and renders, to adisplay, a hybrid representation comprising a particular set of the atleast one 3-D virtual representation, at least a part or portion of theuser, the corresponding interaction, the consequence of thecorresponding interaction, or a combination thereof.

Additional systems and methods for utilizing laboratory equipment in ahybrid reality environment are contemplated herein and include: at leastone first piece of laboratory equipment having at least one feature,wherein the at least one piece of laboratory equipment is tracked; atracking module for tracking a position and an orientation of the atleast one piece of tracked laboratory equipment; a virtual model, storedin a memory, comprising at least one 3-D virtual representation of atleast one feature of an additional piece of laboratory equipment, atleast a portion of a part of a user, or a combination thereof; anexperimentation module that: accesses the virtual model stored in thememory; receives input from the tracking module regarding the positionand the orientation of the at least one first piece of trackedlaboratory equipment; determines, based on the input from the trackingsystem, a corresponding interaction between the at least one piece oflaboratory equipment and at least one additional piece of laboratoryequipment, at least a portion or part of a user, or a combinationthereof; determines a consequence of the corresponding interaction; andrenders, to a display, a hybrid representation comprising the physicalvisual representation of the at least one piece of tracked laboratoryequipment, at least one 3-D virtual representation, at least a part orportion of the user, the corresponding interaction, the consequence ofthe corresponding interaction, or a combination thereof.

A piece of laboratory equipment for use in a hybrid reality environmentor augmented virtual reality environment is also included that comprisesat least one piece of laboratory equipment having at least one feature,wherein the at least one piece of laboratory equipment is tracked; atleast one marker that is coupled with the at least one piece oflaboratory equipment; and a tracking module for tracking a position andan orientation of the at least one piece of tracked laboratoryequipment, wherein the tracking module accesses the at least one marker.

BRIEF DESCRIPTION OF THE FIGURES

Prior Art FIG. 1 shows a comparison of Various Tracking approaches usedin VR/AR/MR/XR technologies. One criteria of interest for this work isthe spatial resolution of the tracking approach.

FIG. 2 includes a diagram showing the various Hardware and Softwarecomponents along with data flow. Active tracking and HMD display usingSteamVR's Lighthouse 2.0 tracking system (left). Passive tracking ofretroreflective markers using an Optitrack IR camera. Small circles showimages of students holding the “virtualized lab tools” we've alreadydeveloped. Green indicated data output. Dashed arrows indicate wirelessor optical data transmission.

FIG. 3 : Illustration of the various “hands-on” VR LabModules/sub-modules to be made and the classes in which they will beimplemented/tested. Blue, red, and green arrows indicate the level(introductory, intermediate, or advanced) at which the module will beimplemented within each class.

FIG. 4 : Each of the three dual-view images in this panel contains aphysical-world (left) and a virtual-world perspective (right) of astudent using virtualized lab tools. The top left dual-view shows thestudent picking up virtual liquid from a passively tracked andvirtualized 250 mL reagent stock container using a passively-tracked andvirtualized pipetteman (in this case a rainin pipette to which we'vemounted retroreflective markers on the pipette shaft, plunger, and tipejector with additional markers 420 included). At bottom left thestudent is transferring the virtual liquid they just picked up into avirtualized 50 mL container. At top right, a student is being shownpouring liquid stock reagent from the passively-tracked 250 mL containerinto an actively-tracked beaker (the Vive Tracker unit 410 is mounted ontop of the beaker).

FIG. 5 : In each class in which the “hands-on” virtual labs are to betested students will be randomly assigned into control-first (CF) andexperiment-first (EF) categories. All students will receive a pre-test,then CF students will be exposed to the control VR module (i.e. eitherwithout authentic tactile sensory feedback from real lab tools orwithout dynamics molecular visualizations) while EF students receive theexperimental VR module. Students will be assessed using real-timeassessment within the VR module but then will perform a mid-test outsideof the VR module as well. Finally, the students will switch activities,additional real-time assessments will be made, and after completion ofthe second version of the lab a post-test will be administered. Thisprocedure will be used for most of the VR modules we will test. For someclasses (i.e. lab classes) additional metrics of student performancewill also be used for assessment (shown at bottom).

FIG. 6 shows contemplated systems and methods.

FIG. 7 shows contemplated systems and methods.

DETAILED DESCRIPTION

Specifically, current contemplated embodiments resolve many of theissues of conventional technologies through the creation and use ofnovel hardware and software, which can provide an authentic tactilesensory experience of doing wet-lab scientific research. In acontemplated embodiment, an “augmented/virtualized—pipetteman” can beused in augmented or virtual reality learning modules for biochemistry,chemistry, and molecular biology courses.

Some additional goals include: developing and utilizing high-precisionmotion capture methods to track the positions, orientations, and statesof the laboratory tools most-often directly manipulated by trainees inchemistry and biochemistry (e.g.

pipettemen, test tubes, beakers, etc.); creating virtual realitychemistry and biochemistry experiences which use this tracking data toprovide students with both real-time macroscopic feedback on theirlaboratory performance as well as just-in-time dynamicmolecular/microscopic visualizations illustrating key concepts relevantto the simulated tasks immediately at hand; and assessing the impact ofsuch targeted-immersion hands-on virtual reality lab experiences onstudent learning, self-efficacy, and intrinsic motivation.

Accordingly, a hybrid virtual student experience has been developedwhere sensors are placed on or built into the items that need to beactually manipulated by the user (e.g. pipettemen and sample tube) inorder to preserve the real feel of the manipulated object while avoidingthe need to replicate the actual device or emulate the force feedbackfrom the virtual world using a haptic glove.

Specifically, systems and methods for utilizing laboratory equipment ina hybrid reality environment, are contemplated herein and include: atleast one piece of laboratory equipment having at least one feature,wherein the at least one piece of laboratory equipment is tracked

Systems and methods for utilizing laboratory equipment 600 in a hybridreality environment, are contemplated herein, are shown in FIG. 6 andinclude: at least one piece of laboratory equipment 610 having at leastone feature, wherein the at least one piece of laboratory equipment istracked; a tracking module 620 for tracking a position and anorientation of the at least one piece of tracked laboratory equipment; avirtual model 630, stored in a memory, comprising at least one 3-Dvirtual representation of the at least one feature of the at least onepiece of laboratory equipment; an experimentation module 640 that:accesses 650 the virtual model stored in the memory; receives 660 inputfrom the tracking module regarding the position and the orientation ofthe at least one piece of tracked laboratory equipment; determines 670,based on the input from the tracking module, a corresponding interactionbetween the at least one piece of laboratory equipment and at least oneadditional piece of laboratory equipment, at least a portion or part ofa user, or a combination thereof; determines 680 a consequence of thecorresponding interaction; and renders 690, to a display, a hybridrepresentation comprising a particular set of the at least one 3-Dvirtual representation, at least a part or portion of the user, thecorresponding interaction, the consequence of the correspondinginteraction, or a combination thereof.

Additional systems and methods 700 for utilizing laboratory equipment ina hybrid reality environment are contemplated herein, shown in FIG. 7 ,and include: at least one first piece of laboratory equipment 710 havingat least one feature, wherein the at least one piece of laboratoryequipment is tracked; a tracking module 720 for tracking a position andan orientation of the at least one piece of tracked laboratoryequipment; a virtual model 730, stored in a memory, comprising at leastone 3-D virtual representation of at least one feature of an additionalpiece of laboratory equipment, at least a portion of a part of a user,or a combination thereof; an experimentation module 740 that: accesses750 the virtual model stored in the memory; receives 760 input from thetracking module regarding the position and the orientation of the atleast one first piece of tracked laboratory equipment; determines 770,based on the input from the tracking system, a corresponding interactionbetween the at least one piece of laboratory equipment and at least oneadditional piece of laboratory equipment, at least a portion or part ofa user, or a combination thereof; determines 780 a consequence of thecorresponding interaction; and renders 790, to a display, a hybridrepresentation comprising the physical visual representation of the atleast one piece of tracked laboratory equipment, at least one 3-Dvirtual representation, at least a part or portion of the user, thecorresponding interaction, the consequence of the correspondinginteraction, or a combination thereof.

A piece of laboratory equipment for use in a hybrid reality environmentor augmented virtual reality environment is also included that comprisesat least one piece of laboratory equipment having at least one feature,wherein the at least one piece of laboratory equipment is tracked; atleast one marker that is coupled with the at least one piece oflaboratory equipment; and a tracking module for tracking a position andan orientation of the at least one piece of tracked laboratoryequipment, wherein the tracking module accessed the at least one marker.

As mentioned throughout this specification, a hybrid reality environmentmeans that real world lab tools, equipment, and components that areconventionally found in academic and industrial labs can be modified orsupplemented so that they can be viewed and manipulated in a virtualenvironment. In some contemplated embodiments, all of the equipment, theuser's hands, the other experimental materials are digitally convertedinto virtual reality. In other contemplated embodiments, some of thecomponents, lab equipment, and user's hands/arms are shown visually asthey physically are in real life, and other components, such as liquids,powders, expensive equipment, hard-to-find materials, or combinationsthereof are shown virtually in the environment, so that the environmentis a hybrid digital/physical environment. However, it should beunderstood that labs, schools, and universities do not need to spendthousands or hundreds of thousands of dollars on special virtual realitytools and equipment (e.g. haptic gloves), but instead they can purchasesensors or other suitable markers/software and utilize equipment thatthey already have on hand together with the billions of “sensors” ornerves that the user will already have in their own hand.

In contemplated embodiments, any of the tools in lab that need to bepicked up, used, or otherwise manipulated by the student, worker, orprofessor can be retrofitted or initially constructed and built withsuitable hardware and devices that allow the actual position and stateof the tools to be detected by a computer and allow transmission ofinformation to a computer or other computing environment that operatesthe virtual reality experience for the user/users.

As used herein, the phrase “at least one piece of laboratory equipmenthaving at least one feature, wherein the at least one piece oflaboratory equipment is tracked” or the phrase “at least one first pieceof laboratory equipment having at least one feature, wherein the atleast one piece of laboratory equipment is tracked” means any suitableor available physical piece of laboratory equipment (or suitablyauthentic replica thereof, such as a piece of laboratory equipmentfabricated from plastic or composite) that someone may use or need to betrained on, including glassware (or plastic replicas of glassware withtracking markers), pipettes, beakers, weighing equipment/scales,analytical equipment and instrumentation, or any other suitable piece ofequipment. In each of these instances, a contemplated piece oflaboratory equipment is tracked by a suitable tracking module. Incontemplated embodiments, this or these pieces of laboratory equipmentmay be actual physical equipment, physical equipment comprisingmaterials other than the original materials, such as plastic orcomposite material, may be virtual equipment, or may be a combinationthereof. As stated earlier, with new students or with expensiveequipment, the equipment may be rendered out of other materials, so thatif the student drops the equipment or handles it too roughly, theequipment won't be broken or lost. In some embodiments, the equipmentmay be 3-D printed, bought, or prepared another way.

In contemplated embodiments, the at least one piece of laboratoryequipment and/or the at least one piece of additional laboratoryequipment may comprise at least one laboratory solid material, at leastone laboratory liquid material, at least one laboratory gaseousmaterial, or a combination thereof, wherein the at least one piece oflaboratory equipment, laboratory solid material, laboratory liquidmaterial, laboratory gaseous material, or a combination thereof is aphysical actual representation, a virtual model representation, or acombination thereof.

Contemplated laboratory solid materials may comprise equipment, but itmay also comprise powders, granules, chunks, crystals, or any othersolid material, chemical or compound found in the laboratory or broughtinto the laboratory for tests or testing. Contemplated laboratory liquidmaterial may comprise any suitable liquid, including water, liquidchemicals, liquid brines or broths, and any other liquid found in thelaboratory or brought into the laboratory for tests or testing.Contemplated gaseous material may comprise any suitable gas or gaseousmaterial that is found in the laboratory or brought into the laboratoryfor tests or testing.

As used herein, the phrase “physical actual representation” meansgenerally a digital representation of the user's actual hand, arm,finger, or other part or portion. It should be understood thatcontemplated systems may show the user a mixture of animated, virtualreality and the user's actual hand, or the actual beaker, or the actualinstrument. The augmented reality may be a blended or mixed view for theuser, depending on the needs of the experiment and the items availablein the laboratory setting.

As used herein, the phrase “a tracking module for tracking a positionand an orientation of the at least one piece of tracked laboratoryequipment” means that a contemplated tracking module has the ability totrack a position and an orientation of at least one piece of trackedlaboratory equipment for the purpose of allowing the user to visualizeit in real time in the hybrid virtual reality environment or space. Acontemplated tracking module may comprise a number of components,including equipment sensors, markers, or transmitters, receivers,processing software, or a combination thereof. As part of thiscontemplated module, physical markers with either non-degenerate markergeometries or digitally encoded identities are utilized by theimaging/tracking software to uniquely recognize each lab tool. Forgeometrically-encoded lab tool identification the it is critical toensure that markers don't look too much like one another, and thus“confuse” the imaging software. Contemplated markers are considered tobe providing either passive or active tracking. Active tracking isdigital and involves using IR sensors as the markers on the lab tools,which works as long as you can provide power to the markers (which meansbulky markers—often preventing utility for tracking smaller lab tools).

As used herein, the phrase “a virtual model, stored in a memory,comprising at least one 3-D virtual representation of at least onefeature of an additional piece of laboratory equipment” means theenvironment where the at least one tracked piece of laboratory equipmentis shown, along with other components of the virtual environment. Theexperiments and some of the equipment and lab components that aredesigned to be fully virtual (not physically located in the actual lab)are also stored in a contemplated virtual model, so that they can beintroduced into the hybrid virtual environment and utilized by the user.A contemplated virtual model may also comprise a portion or a part ofthe user, such as the user's hand/hands, fingers, arm or arms, or othersuitable portions or parts of the user.

In some contemplated embodiments, an experimentation module is utilized,wherein the experimentation module functions to: access the virtualmodel stored in the memory; receive input from the tracking moduleregarding the position and the orientation of the at least one piece oftracked laboratory equipment; determine, based on the input from thetracking system, a corresponding interaction between the at least onepiece of laboratory equipment and at least one additional piece oflaboratory equipment; determine a consequence of the correspondinginteraction; and render, to a display, a hybrid representationcomprising the at least one 3-D virtual representation, at least a partor portion of the user, the corresponding interaction, the consequenceof the corresponding interaction, or a combination thereof.

In other embodiments, a contemplated experimentation module is utilized,wherein the experimentation module functions to: access the virtualmodel stored in the memory; receive input from the tracking moduleregarding the position and the orientation of the at least one firstpiece of tracked laboratory equipment; determine, based on the inputfrom the tracking system, a corresponding interaction between the atleast one piece of laboratory equipment and at least one additionalpiece of laboratory equipment; determine a consequence of thecorresponding interaction; and render, to a display, a hybridrepresentation comprising the physical visual representation of the atleast one piece of tracked laboratory equipment, at least one 3-Dvirtual representation, at least a part or potion of the user, thecorresponding interaction, the consequence of the correspondinginteraction, or a combination thereof

EXAMPLE Augmented-Reality or Virtualized Pipetteman

In one contemplated embodiment, a “virtualized pipetteman” or an“augmented-virtuality pipetteman” comprises a regular pipetteman that isretrofitted or built with hardware (sensors, transmitters, gyroscopes,and accelerometers) capable of detecting and transmitting information onthe status of the pipetteman to an app-based virtual/augmented-realityprogram/learning module. The user's headgear or head-mount display(mobile phone or PC-integrated imaging device) will integrate the actualpipetteman's position, orientation, and status (full of sample, empty,or ejecting liquid) and project an image of this pipetteman into thevirtual reality world being shown to the user through their headgear. Insome contemplated embodiments, there can also be a special test-tube andtip-rack provided, which will allow users to reload virtual tips and putvirtual samples into test tubes that they are actually holding andopening the caps of in their real hands. Contemplated hardware will becompatible with tools/instruments/assets already developed by othertraditional virtual reality companies, as well as any other 3D-basedgaming engine/platform. Drivers and APIs are contemplated that meet thedemands of this contemplated and new technology.

The retrofitted nature of the contemplated pipetteman modification willallow a wide range of pipettes to be “virtualized” enabling a widevariety of different pipettes to be handled and experienced by userscarrying out the virtual experiments and activities. The use of thecontemplated technology will allow more colleges and universities tooffer more sections of science labs without compromising on studentexposure to “hands-on” experimentation—when a virtual but “tactily”authentic version of such experiments is warranted. This will benefitmany molecular biology/biochemistry/chemistry courses and studentsworldwide as they will be able to get real-time feedback on their labprocedure at the level of each individual pipetting step and will beable to perform virtual experiments with authentic tactile feedback asmany times as might be necessary to gain mastery over the methods andprocedures being learned.

Contemplated embodiments include hardware and software that allow astandard real-life pipetteman to be retrofitted and “virtualized” intoan intermediary passable tool that interfaces between studentresearchers and a virtual world filled with other virtualized hands-onlab tools as well as fully virtual laboratory equipment. Users willreceive detailed feedback on every step of the experimental procedurethey perform within the virtual experience. To date, conventionaltechnologies have focused on developing haptic gloves that can emulateinteractions with any fully virtual object at a great hardware cost forthe user and a large decrease in the authenticity of tactile sensoryfeedback provided to the user. Less expensive but narrower-scopeapproaches such as retrofitting particular lab tools with trackingmarkers so that they can both be manipulated naturally by an actualhuman, and so that those manipulations can be tracked and interfacedwith a virtual world of experiments and instruments to provide automatedfeedback to the user on what they might be doing incorrectly in theirexperiments has been lacking. By bridging the gap between the virtualand real worlds in a cost-effective manner and with a narrower scope forthe interactions of interest by “virtualizing” only the importantelements that are most often physically touched by a user (e.g.pipetteman, tubes, beakers, etc), many barriers to student learning canbe minimized or erased altogether.

Improvements in student engagement and a more realistic sensation of howto carry out experiments is needed in many biochemistry, molecularbiology, and chemistry courses. This innovative technology will providethis much-needed resource. Teams of faculty will then be able to buildup the fully or partially-virtual tools/assets required to buildinnovative virtual and augmented instructional laboratory experiencesknowing that students will be manipulating only a small subset of realscientific tools, using them to interface with these entirely orpartially-virtual instruments in specific ways, and also receiving thecompletely realistic “hands-on” tactile sensory feedback from them forwhich traditional labs are so deservedly prized. Teaching laboratoryskills is time-consuming and resource intensive. Students must makemistakes in order to learn but this costs time and money. Augmentedreality and virtual reality systems might well address this howeverthoughtful hardware-software integration is necessary in order toprovide students with an authentic simulated laboratory researchexperience.

The hands-on VR approach we describe herein enables noveltechnology-driven approaches to teaching, learning, and training notonly at universities but also in industry settings. This approach takesactive learning to an entirely new level. We predict that by virtue ofthe decreased cost of hands-on virtual labs all science courses may beable to include effective lab components. The digital assets, methods,and workflows described generally herein will also facilitate futurecontent creation by non-expert programmers and digital artists allowingfaculty in diverse fields to quickly create their own hands-on VRexperiences. Finally, by providing students with additional options forhow and when to consume laboratory course content, such hands-on VR labexperiences may help address the issue of STEM persistence ofunderrepresented minority students-an important national priority.

FIG. 2 includes a diagram 200 showing the various hardware and softwarecomponents along with data flow. Active tracking and head-mount display(HMD) display using SteamVR's Lighthouse 2.0 tracking system (210).Passive tracking of retroreflective markers using an Optitrack IR camera220. Small circles 230 show images of students holding the “virtualizedlab tools” we've already developed. Green indicated data output. Dashedarrows indicate wireless or optical data transmission.

FIG. 3 shows an illustration 300 of the various “hands-on” VR LabModules/sub-modules that are contemplated and the classes in which theywill be implemented/tested. Blue (solid), red (short dash), and green(long, bold dash) arrows indicate the level (introductory, intermediate,or advanced) at which the module will be implemented within each class.

FIG. 4 shows students using simple virtualized lab tools 400. Each ofthe three dual-view images in this panel contains a physical-world(left) and a virtual-world perspective (right) of a student usingvirtualized lab tools. The top left dual-view shows the student pickingup virtual liquid from a passively tracked and virtualized 250 mLreagent stock container using a passively-tracked and virtualizedpipetteman (in this case a rainin pipette to which we've mountedretroreflective markers on the pipette shaft, plunger, and tip ejector).At bottom left the student is transferring the virtual liquid they justpicked up into a virtualized 50 mL container. At top right, a student isbeing shown pouring liquid stock reagent from the passively-tracked 250mL container into an actively-tracked beaker (the Vive Tracker unit ismounted on top of the beaker).

FIG. 5 shows an assessment strategy 500 sequence to be used todemonstrate efficacy of the hands-on virtual labs contemplated. In eachclass in which the “hands-on” virtual labs are to be tested studentswill be randomly assigned into control-first (CF) and experiment-first(EF) categories. All students will receive a pre-test, then CF studentswill be exposed to the control VR module (i.e. either without authentictactile sensory feedback from real lab tools or without dynamicsmolecular visualizations) while EF students receive the experimental VRmodule. Students will be assessed using real-time assessment (RTA)within the VR module but then will perform a mid-test outside of the VRmodule as well. Finally, the students will switch activities, additionalreal-time assessments will be made, and after completion of the secondversion of the lab a post-test will be administered. This procedure willbe used for most of the hands-on VR modules we will test. For someclasses (i.e. lab classes) additional metrics of student performancewill also be used for assessment (shown at bottom).

Thus, specific embodiments and methods of use of hands on laboratory anddemonstration equipment with a hybrid virtual environment have beendisclosed. It should be apparent, however, to those skilled in the artthat many more modifications besides those already described arepossible without departing from the inventive concepts herein. Theinventive subject matter, therefore, is not to be restricted except inthe spirit of the disclosure herein. Moreover, in interpreting thespecification, all terms should be interpreted in the broadest possiblemanner consistent with the context. In particular, the terms “comprises”and “comprising” should be interpreted as referring to elements,components, or steps in a non-exclusive manner, indicating that thereferenced elements, components, or steps may be present, or utilized,or combined with other elements, components, or steps that are notexpressly referenced.

The invention claimed is:
 1. A system for utilizing virtualizedlaboratory equipment in a hybrid reality environment, comprising: atleast two pieces of virtualized laboratory equipment; at least oneoptical tracking module for optical tracking of a position and anorientation of the at least two pieces of virtualized laboratoryequipment, wherein the optical tracking enables a detection of at leastone functionally relevant corresponding interaction between at least twoof the at least two pieces of virtualized laboratory equipment; aplurality of uniquely-identifiable sets of markers that are coupled tothe at least two pieces of virtualized laboratory equipment, wherein themarker comprises at least one non-degenerate marker geometry that ispassively tracked or at least one digital encoded identity that isactively tracked; wherein each piece of virtualized laboratory equipmentis coupled with the plurality of uniquely-identifiable set of markers,and wherein the set of markers and the at least two pieces of laboratoryequipment are entirely optically tracked; a virtual model, stored in amemory, comprising at least two 3-D virtual representations of the twopieces of virtualized laboratory equipment and the plurality ofuniquely-identifiable sets of markers that are coupled to the at leasttwo pieces of virtualized laboratory equipment; an experimentationmodule that: accesses the virtual model stored in the memory; receivesinput from the at least one optical tracking module regarding theposition and the orientation of the at least two pieces of virtualizedlaboratory equipment; determines, based entirely on the input from theat least one optical tracking module, the at least one functionallyrelevant corresponding interaction between the at least one two piecesof virtualized laboratory equipment, at least a portion or part of auser, or a combination thereof; determines a consequence of the at leastone functionally relevant corresponding interaction; and renders, to adisplay, a hybrid representation comprising a particular set of the atleast two 3-D virtual representations, at least a part or portion of theuser, the at least one functionally relevant corresponding interaction,the consequence of the at least one functionally relevant correspondinginteraction, or a combination thereof.
 2. The system of claim 1, furthercomprising at least one piece of laboratory equipment, laboratory solidmaterial, laboratory liquid material, laboratory gaseous material, or acombination thereof, wherein the at least one piece of laboratoryequipment, laboratory solid material, laboratory liquid material,laboratory gaseous material, or a combination thereof is a physicalactual model representation, a virtual model representation or acombination thereof.
 3. The system of claim 2, wherein the at least onepiece of laboratory equipment, laboratory solid material, laboratoryliquid material, laboratory gaseous material, or a combination thereofis a virtual model representation.
 4. The system for utilizingvirtualized laboratory equipment in a hybrid reality environment ofclaim 2, wherein the physical actual model representation, the virtualmodel representation or a combination thereof is a 3D virtual modelrepresentation.
 5. The system of claim 1, further comprising at leastone piece of additional laboratory equipment, laboratory solid material,laboratory liquid material, laboratory gaseous material, or acombination thereof, wherein the at least one piece of additionallaboratory equipment, laboratory solid material, laboratory liquidmaterial, laboratory gaseous material, or a combination thereof is aphysical actual model representation, a virtual model representation ora combination thereof.
 6. The system of claim 5, wherein the at leastone piece of additional laboratory equipment, laboratory solid material,laboratory liquid material, laboratory gaseous material, or acombination thereof is a virtual model representation.
 7. The system forutilizing virtualized laboratory equipment in a hybrid realityenvironment of claim 1, wherein there are at least five pieces ofvirtualized laboratory equipment.
 8. The system for utilizingvirtualized laboratory equipment in a hybrid reality environment ofclaim 1, wherein the at least one uniquely identifiable marker iscontinuously and simultaneously tracked by the at least one trackingmodule.
 9. The system for utilizing virtualized laboratory equipment ina hybrid reality environment of claim 1, wherein the at least oneuniquely identifiable marker is tracked by the at least one trackingmodule with high precision.
 10. The system for utilizing virtualizedlaboratory equipment in a hybrid reality environment of claim 9, whereinhigh precision comprises sub-millimeter resolution.
 11. The system forutilizing virtualized laboratory equipment in a hybrid realityenvironment of claim 1, further comprising a real-time assessmentmodule.
 12. The system for utilizing virtualized laboratory equipment ina hybrid reality environment of claim 11, wherein the real-timeassessment module assesses a user's performance, and providesinformation to the user and an instructor.
 13. The system for utilizingvirtualized laboratory equipment in a hybrid reality environment ofclaim 1, wherein the virtual model in the hybrid reality environmentshows the at least one piece of virtualized laboratory equipment, alongwith at least one other component of a virtual environment.
 14. Thesystem for utilizing virtualized laboratory equipment in a hybridreality environment of claim 1, wherein the at least one piece ofvirtualized laboratory equipment comprises glassware, plastic replicasof glassware, pipettes, beakers, weighing equipment/scales, analyticalequipment, instrumentation, or any other suitable piece of equipment.15. A method of producing virtualized laboratory equipment for use in ahybrid reality environment, comprising: identifying at least two piecesof laboratory equipment to be virtualized; developing a plurality ofuniquely-identifiable sets of markers, wherein each set of markerscomprise at least one non-degenerate marker geometry that can bepassively optically tracked or at least one digital encoded identitythat can be actively optically tracked; providing and utilizing at leastone optical tracking module for optical tracking of a position and anorientation of the at least two pieces of laboratory equipment to bevirtualized and each of their coupled sets of markers, wherein theoptical tracking enables a detection of at least one functionallyrelevant corresponding interaction between at least two of the at leasttwo pieces of laboratory equipment to be virtualized; coupling theplurality of uniquely-identifiable sets of markers to the at least twopieces of laboratory equipment to be virtualized, wherein the pluralityof uniquely-identifiable sets of markers tracked by the optical trackingmodule and wherein the do not interfere with a tactile manipulation anda use of the at least two pieces of laboratory equipment to bevirtualized by a user; creating a virtual model, stored in a memory,comprising at least two 3-D virtual representations of the at least twopieces of laboratory equipment to be virtualized and each of theircoupled uniquely-identifiable sets of markers; utilizing anexperimentation module, wherein the experimentation module functions to:access the virtual model stored in the memory; receive input from the atleast one optical tracking module regarding the position and theorientation of the at least two pieces of laboratory equipment to bevirtualized; determine, based entirely on the input from the at leastone optical tracking module, the at least one functionally relevantcorresponding interaction between the at least two pieces of laboratoryequipment to be virtualized, at least a portion or part of a user, or acombination thereof; determine a consequence of the at least onefunctionally relevant corresponding interaction; and render, to adisplay, a hybrid representation comprising a particular set of the atleast two 3-D virtual representations, at least a part or portion of theuser, the at least one functionally relevant corresponding interaction,the consequence of the at least one functionally relevant correspondinginteraction, or a combination thereof.
 16. The method of claim 15,wherein the plurality of markers comprise passive infraredretroreflectors, active red-green-blue (RGB) emitters, active infraredemitters, active infrared sensors, or a combination thereof.
 17. Themethod of claim 15, wherein the at least one optical tracking modulecomprises at least one motion capture device, at least one motioncapture camera, at least one infrared camera, or a combination thereof.18. A system for utilizing virtualized laboratory equipment in a hybridreality environment, comprising: at least two pieces of virtualizedlaboratory equipment; at least one optical tracking module for opticaltracking of a position and an orientation of the at least two pieces ofvirtualized laboratory equipment, wherein the optical tracking enables adetection of at least one functionally relevant correspondinginteraction between at least two of the at least two pieces ofvirtualized laboratory equipment, and wherein the at least one opticaltracking module and the at least two pieces of virtualized laboratoryequipment are not cost-prohibitive for a science laboratory trainingapplication; a plurality of uniquely-identifiable sets of markers thatare coupled to the at least two pieces of virtualized laboratoryequipment, wherein each set of markers comprises at least onenon-degenerate marker geometry that is passively tracked or at least onedigital encoded identity that is actively tracked, wherein each piece ofvirtualized laboratory equipment is coupled with one set of markers fromthe plurality of uniquely-identifiable sets of markers, wherein the setsof markers and the at least two pieces of laboratory equipment areentirely optically tracked, wherein the coupling minimizes bothinterference with a natural tactile manipulation and use of the at leasttwo pieces of virtualized laboratory equipment and interference with areal-time optical tracking of the sets of markers, and wherein the size,shape, and bulk of the sets of markers minimizes both interference withthe natural tactile manipulation and use of the at least two pieces ofvirtualized laboratory equipment and interference with the real-timeoptical tracking of the sets of markers; a virtual model, stored in amemory, comprising at least two 3-D virtual representations of the twopieces of virtualized laboratory equipment and the plurality ofuniquely-identifiable sets of markers that are coupled to the at leasttwo pieces of virtualized laboratory equipment; an experimentationmodule that: accesses the virtual model stored in the memory; receivesinput from the at least one optical tracking module regarding theposition and the orientation of the at least two pieces of virtualizedlaboratory equipment; determines, based entirely on the input from theat least one optical tracking module, the at least one functionallyrelevant corresponding interaction between the at least two pieces ofvirtualized laboratory equipment, at least a portion or part of a user,or a combination thereof; determines a consequence of the at least onefunctionally relevant corresponding interaction; and renders, to adisplay, a hybrid representation comprising a particular set of the atleast two 3-D virtual representations, at least a part or portion of theuser, the at least one functionally relevant corresponding interaction,the consequence of the at least one functionally relevant correspondinginteraction, or a combination thereof.